Communication optimization strategies for distributed deep neural network training: A survey

Recent trends in high-performance computing and deep learning have led to the proliferation of studies on large-scale neural network training. However, frequent communication requirements among computation nodes drastically slows overall training speeds, which causes bottlenecks distributed training, particularly clusters with limited bandwidths. To mitigate drawbacks communications, researchers proposed various optimization strategies. In this paper, we provide a comprehensive survey strategies from both an algorithm viewpoint computer perspective. Algorithm optimizations focus reducing volumes used while accelerating communications between devices. At level, describe how reduce number rounds transmitted bits per round. addition, elucidate overlap communication. discuss effects caused by infrastructures, including logical schemes protocols. Finally, extrapolate potential future challenges new research directions accelerate for